[Elastic] Add the log collection and diagnosis function by wanglei19991004 · Pull Request #766 · flagos-ai/FlagScale

wanglei19991004 · 2025-09-04T03:01:01Z

Record training status changes in file: outputs/logs/monitor/status.log
incrementally collect log content in file: outputs/logs/monitor/host_*_temp_*.log
automatically analyze training errors and generate report in file: outputs/logs/monitor/host_*_diagnostic_*.txt
diagnostic report examples:

Diagnostic Report for localhost (node 0)
Generated at 2025-09-11 01:53:17
Analysis:
- OutOfMemoryError: The training process ran out of GPU memory. Check host_0_localhost_temp_20250918_163815.log line:32-128
- RendezvousConnectionError: Connection to rendezvous backend failed. Check host_0_localhost_temp_20250918_163815.log line:33-97
- CodeError: Python exception occurred during execution. Check host_0_localhost_temp_20250918_163815.log line:29-125

Add hang detection. If there is no update in the log within 30 minutes, it is determined to be hung
The log monitoring and diagnosis module is disabled by default. Set the switch control:

python run.py action=run +experiment.runner.enable_monitoring=true

or modify train.yaml

experiment:
    runner:
      enable_monitoring: true

gemini-code-assist

Summary of Changes

Hello @wanglei19991004, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability and debugging capabilities for elastic training jobs. It establishes a new framework for systematically gathering training logs from various nodes and automatically generating diagnostic reports, providing immediate insights into potential issues. This system is complemented by a fault injection utility, enabling developers to proactively test the robustness of the logging and diagnostic pipeline.

Highlights

Enhanced Log Collection: Introduced a robust system for incrementally collecting logs from both local and remote (multi-machine) training environments, ensuring efficient and continuous log gathering.
Automated Log Diagnosis: Implemented an automated log diagnosis module that analyzes collected logs against predefined error patterns to identify and categorize common training failures, such as OOM errors, worker errors, and timeouts.
Fault Simulation Utility: Added a new utility to simulate various error conditions by writing specific error keys into log files, enabling comprehensive testing and validation of the log collection and diagnosis system.
Integrated Monitoring: Integrated continuous, asynchronous log monitoring and reporting directly into the training runner, allowing for real-time insights into job status and potential issues, with automatic cleanup upon job completion.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces log collection and diagnostic capabilities for elastic training. It adds new modules for log collection, diagnostics, and fault simulation. The core training runner is updated to use these new features in a background monitoring thread.

The overall approach is good, but there are some critical issues that need to be addressed. The incremental log collection for remote nodes is not implemented correctly and currently downloads the full log file on each check. Additionally, shell commands are constructed without escaping file paths, which poses a security and reliability risk. I've also pointed out some smaller issues like unused variables, non-idiomatic naming, and overly broad exception handling.

I've provided specific suggestions to address these points. After these fixes, the new functionality will be much more robust and efficient.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…resolve the issue of blocking the terminal.

wanglei19991004 · 2025-09-11T09:04:28Z

@gemini-code-assist please review this pull request in details.

gemini-code-assist

Code Review

This pull request introduces a new log collection and diagnostic system, which is a great addition for monitoring and debugging training jobs. The implementation includes a monitoring service, log collector, and diagnostic report generator. However, I've found several critical issues that need to be addressed before merging. These include git conflict markers, a method redefinition bug in runner_train.py, and broken remote log collection logic. There are also some high-severity design issues, such as the use of global state and improper signal handling, which could lead to instability. I've provided detailed comments and suggestions for each issue.

according to bot's review, fix some problem Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

According to bot's review, fix some problems. Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…o fix the manually kill detection logic

… and also fix the manually kill detection logic" This reverts commit c993608.

This reverts commit 75570eb.

…gnosis module is disabled by default

wanglei19991004 · 2025-09-22T03:19:50Z

@gemini-code-assist please review this pull request in details.

gemini-code-assist

Code Review

This pull request introduces a comprehensive log collection and diagnosis system, which is a valuable addition for monitoring training jobs. The implementation includes services for incremental log collection, error diagnosis, hang detection, and manual kill detection. The code is well-structured into separate modules for different functionalities.

My review focuses on several critical issues related to remote execution and file system assumptions, which could cause the monitoring to fail in a distributed, non-shared filesystem environment. I've also identified potential race conditions, inefficiencies, and bugs like method overloading. Additionally, some of the new unit tests contain incorrect assertions that need to be addressed.

I've provided detailed comments and suggestions to resolve these issues. Once these are fixed, this will be a very robust and useful feature.

…d ip detection logic for multiple machine

…'t work

wanglei19991004 added 2 commits September 3, 2025 18:24

add collect log and diagnosis

f0a7591

Fix the blocking bug

c137b4e

wanglei19991004 requested a review from aoyulong as a code owner September 4, 2025 03:01

gemini-code-assist Bot reviewed Sep 4, 2025

View reviewed changes

wanglei19991004 and others added 4 commits September 9, 2025 09:26

Update flagscale/runner/elastic/simulatedFault.py

423a774

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update flagscale/runner/runner_train.py

ba556ac

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Split the log collection and diagnostic logic from SSHTrainRunner to …

6b656f1

…resolve the issue of blocking the terminal.

fix blocking terminal

e657f4f

wanglei19991004 requested a review from a team as a code owner September 11, 2025 06:44

Merge branch 'FlagOpen:main' into elastic

ba6a357

gemini-code-assist Bot reviewed Sep 11, 2025

View reviewed changes

wanglei19991004 and others added 11 commits September 11, 2025 17:37

Update flagscale/runner/elastic/simulatedFault.py

ad90cfb

according to bot's review, fix some problem Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update flagscale/runner/elastic/diagnostic.py

ae59357

According to bot's review, fix some problems. Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

add unin_tests and fix some bugs

03c4ebf

Add the corresponding log line to the diagnostic file, and fix some bugs

4f9fd83

add hang detect, fix diagnostic logic and other bugs

b785c2d

add gpu_health_check

75570eb

provide full version of health check (communication testing), and als…

c993608

…o fix the manually kill detection logic

Revert "provide full version of health check (communication testing),…

84ce042

… and also fix the manually kill detection logic" This reverts commit c993608.

Revert "add gpu_health_check"

5f3829f

This reverts commit 75570eb.

fix the manually kill detection logic, and the log monitoring and dia…

5352a11

…gnosis module is disabled by default

resolve _generate_run_script_train conflict

1060e65

gemini-code-assist Bot reviewed Sep 22, 2025

View reviewed changes

wanglei19991004 added 4 commits September 23, 2025 09:53

fixed based on the review feedback from Gemini Code Assist.

dd33ead

Merge remote-tracking branch 'origin/main' into elastic

4b5ab52

merge elastic test into nvidia

7564090

merge elastic test into nvidia

989f854

merge elastic test into nvidia

279e05e

wanglei19991004 changed the title ~~[elastic] add Log Collection and Diagnosis~~ [Elastic] add Log Collection and Diagnosis Sep 25, 2025

zhaoyinglia and others added 11 commits September 28, 2025 10:26

Merge branch 'main' into elastic

cfcac73

check format

9d08518

remove functional-tests-elastic.yml, fix some unit_test error, and ad…

314fa6e

…d ip detection logic for multiple machine

Adjust code format

9d07afa

Merge branch 'main' into elastic

bee1241

Fix the bug where the enable_monitoring option running autotune doesn…

aafbd96

…'t work

Merge branch 'main' into elastic

0e1508d

delete space

2f106fc

delete space

3366763

Merge branch 'main' into elastic

7e0f8e5

Merge branch 'main' into elastic

ad69e22

aoyulong changed the title ~~[Elastic] add Log Collection and Diagnosis~~ [Elastic] Add the log collection and diagnosis function Sep 30, 2025

aoyulong merged commit b21e9c6 into flagos-ai:main Sep 30, 2025
24 of 30 checks passed

Uh oh!

Conversation

wanglei19991004 commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wanglei19991004 commented Sep 11, 2025

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wanglei19991004 commented Sep 22, 2025

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wanglei19991004 commented Sep 4, 2025 •

edited

Loading